Analyzing Margins in Boosting

While the success of boosting or voting methods has been evident from experimental data [11], questions about why boosting does not overfit on training data remain. One idea about the effectiveness of boosting was given by Schapire et al. in which they presented an explanation of why the test error of a boosting classifier does not increase with its size by looking at margins. They showed that boosting increases the margins of training examples effectively and that this increase in margins is related to the auspicious performance of AdaBoost [12]. Breiman invented ArcGv, an algorithm that increases margins even more aggressively than AdaBoost, yet does not perform better. He claimed this contradicts Schapire’s margins explanation, since all else being equal a higher margins distribution did not in this case indicate better results [1]. In this paper, we experimentally examine Breiman’s results by comparing the margins and performance of AdaBoost and ArcGv. We also analyze our approaches and show that our results partially contradict Breiman’s findings. We also explore some other voting algorithms’ performance and present some studies only indirectly related to this problem. We also present possibilities for further research and experimentation in this area, which it is our intention to continue pursuing.